{
  "cells": [
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "copyright"
      },
      "outputs": [],
      "source": [
        "# Copyright 2021 Google LLC\n",
        "#\n",
        "# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
        "# you may not use this file except in compliance with the License.\n",
        "# You may obtain a copy of the License at\n",
        "#\n",
        "#     https://www.apache.org/licenses/LICENSE-2.0\n",
        "#\n",
        "# Unless required by applicable law or agreed to in writing, software\n",
        "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
        "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
        "# See the License for the specific language governing permissions and\n",
        "# limitations under the License."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "JAPoU8Sm5E6e"
      },
      "source": [
        "<table align=\"left\">\n",
        "\n",
        "  <td>\n",
        "    <a href=\"https://colab.research.google.com/github/GoogleCloudPlatform/vertex-ai-samples/blob/master/notebooks/official/automl/automl-tabular-classification.ipynb\"\">\n",
        "      <img src=\"https://cloud.google.com/ml-engine/images/colab-logo-32px.png\" alt=\"Colab logo\"> Run in Colab\n",
        "    </a>\n",
        "  </td>\n",
        "  <td>\n",
        "    <a href=\"https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/master/notebooks/official/automl/automl-tabular-classification.ipynb\">\n",
        "      <img src=\"https://cloud.google.com/ml-engine/images/github-logo-32px.png\" alt=\"GitHub logo\">\n",
        "      View on GitHub\n",
        "    </a>\n",
        "  </td>\n",
        "</table>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "tvgnzT1CKxrO"
      },
      "source": [
        "# Vertex AI Model Builder SDK: AutoML Tabular Training and Prediction\n",
        "\n",
        "To use this Colaboratory notebook, you copy the notebook to your own Google Drive and open it with Colaboratory (or Colab). You can run each step, or cell, and see its results. To run a cell, use Shift+Enter. Colab automatically displays the return value of the last line in each cell. For more information about running notebooks in Colab, see the Colab welcome page.\n",
        "\n",
        "\n",
        "## Overview\n",
        "\n",
        "This tutorial demonstrates how to use the Vertex AI Python client library to train and deploy a tabular classification model for online prediction.\n",
        "\n",
        "**Note**: you may incur charges for training, prediction, storage, or usage of other GCP products in connection with testing this SDK.\n",
        "\n",
        "### Dataset\n",
        "\n",
        "The dataset we are using is the PetFinder Dataset, available locally in Colab. To learn more about this dataset, visit https://www.kaggle.com/c/petfinder-adoption-prediction.\n",
        "\n",
        "### Objective\n",
        "\n",
        "This notebook demonstrates, using the Vertex AI Python client library, how to train and make predictions on an AutoML model based on a tabular dataset. Alternatively, you can train and make predictions on models by using the `gcloud` command-line tool or by using the online Cloud Console.\n",
        "\n",
        "The steps performed include the following:\n",
        "\n",
        "- Create a Vertex AI model training job.\n",
        "- Train an AutoML Tabular model.\n",
        "- Deploy the `Model` resource to a serving `Endpoint` resource.\n",
        "- Make a prediction by sending data.\n",
        "- Undeploy the `Model` resource.\n",
        "\n",
        "### Costs \n",
        "\n",
        "This tutorial uses billable components of Google Cloud:\n",
        "\n",
        "* Vertex AI\n",
        "* Cloud Storage\n",
        "\n",
        "Learn about [Vertex AI\n",
        "pricing](https://cloud.google.com/vertex-ai/pricing) and [Cloud Storage\n",
        "pricing](https://cloud.google.com/storage/pricing), and use the [Pricing\n",
        "Calculator](https://cloud.google.com/products/calculator/)\n",
        "to generate a cost estimate based on your projected usage."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "install_aip"
      },
      "source": [
        "## Installation"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "PyQmSRbKA8r-"
      },
      "outputs": [],
      "source": [
        "import os\n",
        "\n",
        "# The Google Cloud Notebook product has specific requirements\n",
        "IS_GOOGLE_CLOUD_NOTEBOOK = os.path.exists(\"/opt/deeplearning/metadata/env_version\")\n",
        "\n",
        "USER_FLAG = \"\"\n",
        "# Google Cloud Notebook requires dependencies to be installed with '--user'\n",
        "if IS_GOOGLE_CLOUD_NOTEBOOK:\n",
        "    USER_FLAG = \"--user\""
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "b03b7f4487ff"
      },
      "source": [
        "Install the latest version of the Vertex AI client library.\n",
        "\n",
        "Run the following command in your virtual environment to install the Vertex SDK for Python:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "d489d38261dd"
      },
      "outputs": [],
      "source": [
        "! pip install {USER_FLAG} --upgrade google-cloud-aiplatform"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "install_storage"
      },
      "source": [
        "Install the Cloud Storage library:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "qssss-KSlugo"
      },
      "outputs": [],
      "source": [
        "! pip install {USER_FLAG} --upgrade google-cloud-storage"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "d3a26cb9b19d"
      },
      "source": [
        "### Restart the kernel\n",
        "\n",
        "After you install the additional packages, you need to restart the notebook kernel so it can find the packages."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "EzrelQZ22IZj"
      },
      "outputs": [],
      "source": [
        "# Automatically restart kernel after installs\n",
        "import os\n",
        "\n",
        "if not os.getenv(\"IS_TESTING\"):\n",
        "    # Automatically restart kernel after installs\n",
        "    import IPython\n",
        "\n",
        "    app = IPython.Application.instance()\n",
        "    app.kernel.do_shutdown(True)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "lWEdiXsJg0XY"
      },
      "source": [
        "### Select a GPU runtime\n",
        "\n",
        "**Make sure you're running this notebook in a GPU runtime if you have that option. In Colab, select \"Runtime --> Change runtime type > GPU\"**"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "BF1j6f9HApxa"
      },
      "source": [
        "### Set up your Google Cloud project\n",
        "\n",
        "**The following steps are required, regardless of your notebook environment.**\n",
        "\n",
        "1. [Select or create a Google Cloud project](https://console.cloud.google.com/cloud-resource-manager). When you first create an account, you get a $300 free credit towards your compute/storage costs.\n",
        "\n",
        "1. [Make sure that billing is enabled for your project](https://cloud.google.com/billing/docs/how-to/modify-project).\n",
        "\n",
        "1. [Enable the Vertex AI API and Compute Engine API](https://console.cloud.google.com/flows/enableapi?apiid=aiplatform.googleapis.com,compute_component).\n",
        "\n",
        "1. If you are running this notebook locally, you will need to install the [Cloud SDK](https://cloud.google.com/sdk).\n",
        "\n",
        "1. Enter your project ID in the cell below. Then run the cell to make sure the\n",
        "Cloud SDK uses the right project for all the commands in this notebook.\n",
        "\n",
        "**Note**: Jupyter runs lines prefixed with `!` as shell commands, and it interpolates Python variables prefixed with `$` into these commands."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "WReHDGG5g0XY"
      },
      "source": [
        "### Set your project ID\n",
        "\n",
        "**If you don't know your project ID**, you may be able to get your project ID using `gcloud`."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "oM1iC_MfAts1"
      },
      "outputs": [],
      "source": [
        "PROJECT_ID = \"\"\n",
        "\n",
        "# Get your Google Cloud project ID from gcloud\n",
        "if not os.getenv(\"IS_TESTING\"):\n",
        "    shell_output=!gcloud config list --format 'value(core.project)' 2>/dev/null\n",
        "    PROJECT_ID = shell_output[0]\n",
        "    print(\"Project ID: \", PROJECT_ID)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "qJYoRfYng0XZ"
      },
      "source": [
        "Otherwise, set your project ID here."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "riG_qUokg0XZ"
      },
      "outputs": [],
      "source": [
        "if PROJECT_ID == \"\" or PROJECT_ID is None:\n",
        "    PROJECT_ID = \"[your-project-id]\"  # @param {type:\"string\"}"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "06571eb4063b"
      },
      "source": [
        "### Timestamp\n",
        "\n",
        "If you are in a live tutorial session, you might be using a shared test account or project. To avoid name collisions between users on resources created, you create a timestamp for each instance session, and append it onto the name of resources you create in this tutorial."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "697568e92bd6"
      },
      "outputs": [],
      "source": [
        "from datetime import datetime\n",
        "\n",
        "TIMESTAMP = datetime.now().strftime(\"%Y%m%d%H%M%S\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "dr--iN2kAylZ"
      },
      "source": [
        "### Authenticate your Google Cloud account\n",
        "\n",
        "**If you are using Notebooks**, your environment is already\n",
        "authenticated. Skip this step."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "sBCra4QMA2wR"
      },
      "source": [
        "**If you are using Colab**, run the cell below and follow the instructions\n",
        "when prompted to authenticate your account via oAuth.\n",
        "\n",
        "**Otherwise**, follow these steps:\n",
        "\n",
        "1. In the Cloud Console, go to the [**Create service account key**\n",
        "   page](https://console.cloud.google.com/apis/credentials/serviceaccountkey).\n",
        "\n",
        "2. Click **Create service account**.\n",
        "\n",
        "3. In the **Service account name** field, enter a name, and\n",
        "   click **Create**.\n",
        "\n",
        "4. In the **Grant this service account access to project** section, click the **Role** drop-down list. Type \"Vertex AI\"\n",
        "into the filter box, and select\n",
        "   **Vertex AI Administrator**. Type \"Storage Object Admin\" into the filter box, and select **Storage Object Admin**.\n",
        "\n",
        "5. Click *Create*. A JSON file that contains your key downloads to your\n",
        "local environment.\n",
        "\n",
        "6. Enter the path to your service account key as the\n",
        "`GOOGLE_APPLICATION_CREDENTIALS` variable in the cell below and run the cell."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "zgPO1eR3CYjk"
      },
      "source": [
        "### Create a Cloud Storage bucket\n",
        "\n",
        "**The following steps are required, regardless of your notebook environment.**\n",
        "\n",
        "This notebook demonstrates how to use Model Builder SDK to create an AutoML model based on a tabular dataset. You will need to provide a Cloud Storage bucket where the dataset will be stored.\n",
        "\n",
        "Set the name of your Cloud Storage bucket below. It must be unique across all of your \n",
        "Cloud Storage buckets.\n",
        "\n",
        "You may also change the `REGION` variable, which is used for operations\n",
        "throughout the rest of this notebook. Make sure to [choose a region where Vertex AI services are\n",
        "available](https://cloud.google.com/vertex-ai/docs/general/locations). You may\n",
        "not use a Multi-Regional Storage bucket for training with Vertex AI."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "MzGDU7TWdts_"
      },
      "outputs": [],
      "source": [
        "BUCKET_NAME = \"gs://[your-bucket-name]\"  # @param {type:\"string\"}\n",
        "REGION = \"[your-region]\"  # @param {type:\"string\"}"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "cf221059d072"
      },
      "outputs": [],
      "source": [
        "if BUCKET_NAME == \"\" or BUCKET_NAME is None or BUCKET_NAME == \"gs://[your-bucket-name]\":\n",
        "    BUCKET_NAME = \"gs://\" + PROJECT_ID + \"aip-\" + TIMESTAMP"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "-EcIXiGsCePi"
      },
      "source": [
        "**Only if your bucket doesn't already exist**: Run the following cell to create your Cloud Storage bucket."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "NIq7R4HZCfIc"
      },
      "outputs": [],
      "source": [
        "! gsutil mb -l $REGION $BUCKET_NAME"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ucvCsknMCims"
      },
      "source": [
        "Finally, validate access to your Cloud Storage bucket by examining its contents:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "vhOb7YnwClBb"
      },
      "outputs": [],
      "source": [
        "! gsutil ls -al $BUCKET_NAME"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "d89a66b8923f"
      },
      "source": [
        "### Copy dataset into your Cloud Storage bucket"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "59a46204cddb"
      },
      "outputs": [],
      "source": [
        "IMPORT_FILE = \"petfinder-tabular-classification.csv\"\n",
        "! gsutil cp gs://cloud-samples-data/ai-platform-unified/datasets/tabular/{IMPORT_FILE} {BUCKET_NAME}/data/\n",
        "\n",
        "gcs_source = f\"{BUCKET_NAME}/data/{IMPORT_FILE}\""
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Y9Uo3tifg1kx"
      },
      "source": [
        "### Import Vertex SDK for Python\n",
        "\n",
        "Import the Vertex SDK into your Python environment and initialize it."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "pRUOFELefqf1"
      },
      "outputs": [],
      "source": [
        "import os\n",
        "\n",
        "from google.cloud import aiplatform\n",
        "\n",
        "aiplatform.init(project=PROJECT_ID, location=REGION)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "643dfd86b00d"
      },
      "source": [
        "## Tutorial\n",
        "\n",
        "Now you are ready to create your AutoML Tabular model."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8f4f50a0112c"
      },
      "source": [
        "### Create a Managed Tabular Dataset from a CSV\n",
        "\n",
        "This section will create a dataset from a CSV file stored on your GCS bucket."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "f1eef64ee47b"
      },
      "outputs": [],
      "source": [
        "ds = dataset = aiplatform.TabularDataset.create(\n",
        "    display_name=\"petfinder-tabular-dataset\",\n",
        "    gcs_source=gcs_source,\n",
        ")\n",
        "\n",
        "ds.resource_name"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ba5011d50ac7"
      },
      "source": [
        "### Launch a Training Job to Create a Model\n",
        "\n",
        "Once we have defined your training script, we will create a model. The `run` function creates a training pipeline that trains and creates a `Model` object. After the training pipeline completes, the `run` function returns the `Model` object."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "24c2c081d683"
      },
      "outputs": [],
      "source": [
        "job = aiplatform.AutoMLTabularTrainingJob(\n",
        "    display_name=\"train-petfinder-automl-1\",\n",
        "    optimization_prediction_type=\"classification\",\n",
        "    column_transformations=[\n",
        "        {\"categorical\": {\"column_name\": \"Type\"}},\n",
        "        {\"numeric\": {\"column_name\": \"Age\"}},\n",
        "        {\"categorical\": {\"column_name\": \"Breed1\"}},\n",
        "        {\"categorical\": {\"column_name\": \"Color1\"}},\n",
        "        {\"categorical\": {\"column_name\": \"Color2\"}},\n",
        "        {\"categorical\": {\"column_name\": \"MaturitySize\"}},\n",
        "        {\"categorical\": {\"column_name\": \"FurLength\"}},\n",
        "        {\"categorical\": {\"column_name\": \"Vaccinated\"}},\n",
        "        {\"categorical\": {\"column_name\": \"Sterilized\"}},\n",
        "        {\"categorical\": {\"column_name\": \"Health\"}},\n",
        "        {\"numeric\": {\"column_name\": \"Fee\"}},\n",
        "        {\"numeric\": {\"column_name\": \"PhotoAmt\"}},\n",
        "    ],\n",
        ")\n",
        "\n",
        "# This will take around an hour to run\n",
        "model = job.run(\n",
        "    dataset=ds,\n",
        "    target_column=\"Adopted\",\n",
        "    training_fraction_split=0.8,\n",
        "    validation_fraction_split=0.1,\n",
        "    test_fraction_split=0.1,\n",
        "    model_display_name=\"adopted-prediction-model\",\n",
        "    disable_early_stopping=False,\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "93a4d034c73b"
      },
      "source": [
        "### Deploy your model\n",
        "\n",
        "Before you use your model to make predictions, you need to deploy it to an `Endpoint`. You can do this by calling the `deploy` function on the `Model` resource. This function does two things:\n",
        "\n",
        "1. Creates an `Endpoint` resource to which the `Model` resource will be deployed.\n",
        "2. Deploys the `Model` resource to the `Endpoint` resource.\n",
        "\n",
        "Deploy your model.\n",
        "\n",
        "### NOTE: Wait until the model **FINISHES** deployment before proceeding to prediction."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "a371544057d9"
      },
      "outputs": [],
      "source": [
        "endpoint = model.deploy(\n",
        "    machine_type=\"n1-standard-4\",\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "fd44380b9ae3"
      },
      "source": [
        "### Predict on the endpoint\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "13f3e8aa27c0"
      },
      "source": [
        "* This sample instance is taken from an observation in which `Adopted` = **Yes**\n",
        "* Note that the values are all strings. Since the original data was in CSV format, everything is treated as a string. The transformations you defined when creating your `AutoMLTabularTrainingJob` inform Vertex AI to transform the inputs to their defined types.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "00c0d01dc8ae"
      },
      "outputs": [],
      "source": [
        "prediction = endpoint.predict(\n",
        "    [\n",
        "        {\n",
        "            \"Type\": \"Cat\",\n",
        "            \"Age\": \"3\",\n",
        "            \"Breed1\": \"Tabby\",\n",
        "            \"Gender\": \"Male\",\n",
        "            \"Color1\": \"Black\",\n",
        "            \"Color2\": \"White\",\n",
        "            \"MaturitySize\": \"Small\",\n",
        "            \"FurLength\": \"Short\",\n",
        "            \"Vaccinated\": \"No\",\n",
        "            \"Sterilized\": \"No\",\n",
        "            \"Health\": \"Healthy\",\n",
        "            \"Fee\": \"100\",\n",
        "            \"PhotoAmt\": \"2\",\n",
        "        }\n",
        "    ]\n",
        ")\n",
        "\n",
        "print(prediction)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "154258dfb12f"
      },
      "source": [
        "### Undeploy the model\n",
        "\n",
        "To undeploy your `Model` resource from the serving `Endpoint` resource, use the endpoint's `undeploy` method with the following parameter:\n",
        "\n",
        "- `deployed_model_id`: The model deployment identifier returned by the prediction service when the `Model` resource is deployed. You can retrieve the `deployed_model_id` using the prediction object's `deployed_model_id` property."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "186856f896fc"
      },
      "outputs": [],
      "source": [
        "endpoint.undeploy(deployed_model_id=prediction.deployed_model_id)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "d7d2aa967f46"
      },
      "source": [
        "# Cleaning up\n",
        "\n",
        "To clean up all Google Cloud resources used in this project, you can [delete the Google Cloud project](https://cloud.google.com/resource-manager/docs/creating-managing-projects#shutting_down_projects) you used for the tutorial.\n",
        "\n",
        "Otherwise, you can delete the individual resources you created in this tutorial:\n",
        "\n",
        "- Training Job\n",
        "- Model\n",
        "- Endpoint\n",
        "- Cloud Storage Bucket\n",
        "\n",
        "**Note**: You must delete any `Model` resources deployed to the `Endpoint` resource before deleting the `Endpoint` resource."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "1a9c201f8589"
      },
      "outputs": [],
      "source": [
        "delete_training_job = True\n",
        "delete_model = True\n",
        "delete_endpoint = True\n",
        "\n",
        "# Warning: Setting this to true will delete everything in your bucket\n",
        "delete_bucket = False\n",
        "\n",
        "# Delete the training job\n",
        "job.delete()\n",
        "\n",
        "# Delete the model\n",
        "model.delete()\n",
        "\n",
        "# Delete the endpoint\n",
        "endpoint.delete()\n",
        "\n",
        "if delete_bucket and \"BUCKET_NAME\" in globals():\n",
        "    ! gsutil -m rm -r $BUCKET_NAME"
      ]
    }
  ],
  "metadata": {
    "colab": {
      "collapsed_sections": [],
      "name": "automl-tabular-classification.ipynb",
      "toc_visible": true
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}
